Imaging Method

ABSTRACT

A wearable computing device or a head-mounted display (HMD) may be configured to track the gaze axis of an eye of the wearer. In particular, the device may be configured to observe movement of a wearer&#39;s pupil and, based on the movement, determine inputs to a user interface. For example, using eye gaze detection, the HMD may change a tracking rate of a displayed virtual image based on where the user is looking. Gazing at the center of the HMD field of view may, for instance, allow for fine movements of the virtual display. Gazing near an edge of the HMD field of view may provide coarser movements.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a division of U.S. patent application Ser.No. 13/287,390, filed Nov. 2, 2011, which is hereby incorporated byreference into the present application.

BACKGROUND

Wearable systems can integrate various elements, such as miniaturizedcomputers, input devices, sensors, detectors, image displays, wirelesscommunication devices as well as image and audio processors, into adevice that can be worn by a user. Such devices provide a mobile andlightweight solution to communicating, computing and interacting withone's environment. With the advance of technologies associated withwearable systems and miniaturized optical elements, it has becomepossible to consider wearable compact optical displays that augment thewearer's experience of the real world.

By placing an image display element close to the wearer's eye(s), anartificial image can be made to overlay the wearer's view of the realworld. Such image display elements are incorporated into systems alsoreferred to as “near-eye displays”, “head-mounted displays” (HMDs) or“heads-up displays” (HUDs). Depending upon the size of the displayelement and the distance to the wearer's eye, the artificial image mayfill or nearly fill the wearer's field of view.

SUMMARY

In a first aspect, a head-mounted display (HMD) is provided. The HMDincludes a head-mounted support, an optical system, an infrared lightsource, a camera, and a computer. The optical system is attached to thehead-mounted support and includes a display panel configured to generatea virtual image, wherein the virtual image is viewable from a viewinglocation. The infrared light source is configured to illuminate theviewing location with infrared light such that infrared light isreflected from the viewing location as reflected infrared light and thecamera is configured to image the viewing location by collecting thereflected infrared light. The computer is configured to determine a gazeaxis based on one or more images of the viewing location obtained by thecamera and control the display panel to move the virtual images within afield of view based on the gaze axis, a reference axis related to theHMD, and a tracking rate.

In a second aspect, a method is provided. The method includesdetermining a gaze axis within a field of view of a head-mounted display(HMD), wherein the HMD is configured to display virtual images withinthe field of view. The method further includes determining a referenceaxis related to the HMD, adjusting a tracking rate based on the gazeaxis and the reference axis, and moving the virtual images within thefield of view based on the gaze axis, the reference axis and thetracking rate.

In a third aspect, a non-transitory computer readable medium isprovided. The non-transitory computer readable medium includesinstructions executable by a computing device to cause the computingdevice to perform functions including, receiving eye-tracking imagesfrom a head-mounted display (HMD), wherein the HMD is configured todisplay virtual images within a field of view. The non-transitorycomputer readable medium further includes determining a gaze axis fromthe eye-tracking images, determining a reference axis related to theHMD, calculating an angle difference between the gaze axis and thereference axis, and adjusting a tracking rate based on the angledifference. The non-transitory computer readable medium further includescontrolling the HMD to display the virtual images based upon the gazeaxis, the reference axis, and the tracking rate.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is schematic diagram of a wearable computing device, inaccordance with an example embodiment.

FIG. 2 is a top view of an optical system, in accordance with an exampleembodiment.

FIG. 3A is a front view of a head-mounted display, in accordance with anexample embodiment.

FIG. 3B is a top view of the head-mounted display of FIG. 3A, inaccordance with an example embodiment.

FIG. 3C is a side view of the head-mounted display of FIG. 3A and FIG.3B, in accordance with an example embodiment.

FIG. 4A is a side view of a head-mounted display with a forward gazeaxis, in accordance with an example embodiment.

FIG. 4B is a side view of the head-mounted display of FIG. 4A with anupward gaze axis, in accordance with an example embodiment.

FIG. 5 is a flowchart of a method, in accordance with an exampleembodiment.

FIG. 6A is a field of view of a head-mounted display showing scrollingtext, in accordance with an example embodiment.

FIG. 6B is a field of view of a head-mounted display showing scrollingtext, in accordance with an example embodiment.

FIG. 6C is a field of view of a head-mounted display showing scrollingtext, in accordance with an example embodiment.

FIG. 6D is a field of view of a head-mounted display showing scrollingtext, in accordance with an example embodiment.

FIG. 7A is an overhead view of HMD user traveling on a subway, inaccordance with an example embodiment.

FIG. 7B is a field of view of a HMD user interface, in accordance withan example embodiment.

FIG. 7C is a field of view of a HMD user interface, in accordance withan example embodiment.

FIG. 7D is a field of view of a HMD user interface, in accordance withan example embodiment.

FIG. 7E is a field of view of a HMD user interface, in accordance withan example embodiment.

DETAILED DESCRIPTION

In the following detailed description, reference is made to theaccompanying figures, which form a part thereof. In the figures, similarsymbols typically identify similar components, unless context dictatesotherwise. The illustrative embodiments described in the detaileddescription and figures are not meant to be limiting. Other embodimentsmay be utilized, and other changes may be made, without departing fromthe spirit or scope of the subject matter presented herein. It will bereadily understood that the aspects of the present disclosure, asgenerally described herein, and illustrated in the figures, can bearranged, substituted, combined, separated, and designed in a widevariety of different configurations, all of which are contemplatedherein.

1. Overview

A head-mounted display (HMD) may enable its wearer to observe thewearer's real-world surroundings and also view a displayed image, suchas a computer-generated image. In some cases, the displayed image mayoverlay a portion of the wearer's field of view of the real world. Thus,while the wearer of the HMD is going about his or her daily activities,such as walking, driving, exercising, etc., the wearer may be able tosee a displayed image generated by the HMD at the same time that thewearer is looking out at his or her real-world surroundings.

The displayed image, also known as a virtual image, might include, forexample, graphics, text, and/or video. The content of the displayedimage could relate to any number of contexts, including but not limitedto the wearer's current environment, an activity in which the wearer iscurrently engaged, the biometric status of the wearer, and any audio,video, or textual communications that have been directed to the wearer.The images displayed by the HMD may also be part of an interactive userinterface. For example, the HMD could be part of a wearable computingdevice. Thus, the images displayed by the HMD could include menus,selection boxes, navigation icons, or other user interface features thatenable the wearer to invoke functions of the wearable computing deviceor otherwise interact with the wearable computing device.

The images displayed by the HMD could appear anywhere in the wearer'sfield of view. For example, the displayed image might occur at or nearthe center of the wearer's field of view, or the displayed image mightbe confined to the top, bottom, or a corner of the wearer's field ofview. Alternatively, the displayed image might be at the periphery of orentirely outside of the wearer's normal field of view. For example, thedisplayed image might be positioned such that it is not visible when thewearer looks straight ahead but is visible when the wearer looks in aspecific direction, such as up, down, or to one side. In addition, thedisplayed image might overlay only a small portion of the wearer's fieldof view, or the displayed image might fill most or all of the wearer'sfield of view. The displayed image could be displayed continuously oronly at certain times (e.g., only when the wearer is engaged in certainactivities).

The displayed images may appear fixed relative to the wearer'senvironment. For instance, the virtual images may appear anchored to aparticular object or location within the wearer's environment.Alternatively, displayed images may appear fixed relative to thewearer's field of view. For example, the HMD may include a graphicaluser interface that may stay substantially anchored to the wearer'sfield of view regardless of the HMD orientation. In practice, both typesof virtual imagery may be implemented together in an HMD.

To display a virtual image to the wearer, an optical system in the HMDmay include a light source, such as a light-emitting diode (LED), thatis configured to illuminate a display panel, such as a liquidcrystal-on-silicon (LCOS) display. The display panel generates lightpatterns by spatially modulating the light from the light source, andthe light patterns may be viewable as virtual images at a viewinglocation.

The HMD may obtain data from the wearer in order to perform certainfunctions, for instance to provide context-sensitive information to thewearer. In an example embodiment, the HMD may obtain informationregarding the wearer and the wearer's environment and respondaccordingly. For instance, the HMD may use a pupil position recognitiontechnique, wherein if the HMD recognizes that the wearer's pupillocation, and thus a corresponding gaze axis, is inclined with respectto a reference axis, the HMD may display virtual images related toobjects located above the wearer. Conversely, the HMD may recognize, bya similar pupil position recognition technique, that the wearer islooking downward. Accordingly, the HMD may display virtual imagesrelated to objects located below a reference axis of the wearer.

In order to determine the actual position of a HMD wearer's pupil and todetermine a corresponding gaze axis, the wearer's pupil may beilluminated by an infrared light source or multiple infrared lightsources. An infrared camera may image the pupil. The infrared lightsource(s) could be located in the HMD optical path, or couldalternatively be located off-axis. The infrared camera could also belocated in the HMD optical path or off-axis. Possible eye trackingmodalities that could be used include dark pupil imaging and dual-glintPurkinje image tracking, among other techniques known in the art.

A processor may implement an image processing algorithm to find theedges or extents of the imaged pupil. The image processing algorithmsmay include pattern recognition, Canny edge detection, thresholding,contrast detection, or differential edge detection, to name a few. Thoseskilled in the art will understand that a variety of different imageprocessing techniques could be used individually or in combination withother methods in order to obtain pupil location. After image processing,the processor may determine a gaze axis, which may be defined as an axisextending from a viewing location and through a gaze point locatedwithin the wearer's field of view.

The processor may also determine a reference axis, which may be definedas an axis extending from a viewing location and through a point inspace. The point in space may include the apparent center of the displayof the HMD or a target object, among other possibilities.

Once a gaze axis is determined, the processor may act to adjust variouscomponents of the displayed virtual image based on an angle differencebetween the gaze axis and the reference axis and a tracking rate. Forexample, an upward scrolling list of text information may be presentedto a user of an HMD, similar to the traditional display of credits atthe end of a movie. In this embodiment, the reference axis may includean axis that extends through the apparent center of the HMD display. Ifa gaze axis is determined that indicates the user is gazing down nearthe bottom of the display with respect to the reference axis, thetracking rate may be increased such that the rate of upward scrolling isincreased, providing more text to the user.

Conversely, if a gaze axis is determined that indicates that the user isgazing near the top of the screen with respect to the reference axis,the tracking rate may be decreased such that the rate of upwardscrolling is slowed, allowing the user to catch up in reading the text.

Alternatively or additionally, if a gaze axis is determined to be nearthe reference axis (i.e. the user is looking at the middle of thescreen), the tracking rate may be decreased or set to zero (eliminatingscrolling of the virtual image). In this respect, a user may be able tofocus his/her attention on specific text.

It will be evident to those skilled in the art that there are a varietyof ways to implement such virtual image adjustment in a HMD system. Thedetails of such implementations may depend on, for example, the type ofdata provided, the local environmental conditions, the location of theuser, and the task to be performed.

Certain illustrative examples of using eye-tracking data to adjust avirtual image displayed by a HMD are described below. It is to beunderstood, however, that other embodiments are possible and areimplicitly considered within the context of the following exampleembodiments.

2. Adjusting Virtual Images in a Head-Mounted Display Using Eye-Tracking

FIG. 1 is schematic diagram of a wearable computing device or ahead-mounted display (HMD) 100 that may include several differentcomponents and subsystems. In one example, HMD 100 includes aneye-tracking system 102, a HMD-tracking system 104, an optical system106, peripherals, a power supply 110, a processor 112, a memory 114, anda user interface 115. The eye-tracking system 102 may include hardwaresuch as an infrared camera 116 and at least one infrared light source118. The HMD-tracking system 104 may include a gyroscope 120, a globalpositioning system (GPS) 122, and an accelerometer 124. The opticalsystem 106 may include, in one embodiment, a display panel 126, adisplay light source 128, and optics 130. Peripherals 108 may include,for example, a wireless communication interface 134, a touchpad 136, amicrophone 138, a camera 140, and a speaker 142.

In an example embodiment, HMD 100 includes a see-through display. Thus,the wearer of HMD 100 may observe a portion of the real-worldenvironment, i.e., in a particular field of view provided by the opticalsystem 106. In addition, HMD 100 is operable to display virtual imagesthat are superimposed on the field of view, for example, to provide an“augmented reality” experience. Some of the virtual images displayed byHMD 100 may be superimposed over particular objects in the field ofview. HMD 100 may also display images that appear to hover within thefield of view instead of being associated with particular objects in thefield of view.

Components of the HMD 100 may be configured to work in an interconnectedfashion with other components within or outside their respectivesystems. For instance, in an example embodiment, the infrared camera 116may image one or both of the HMD wearer's eyes. The infrared camera 116may deliver image information to the processor 112, which may access thememory 112 and make a determination regarding the direction of the HMDwearer's gaze, also termed a gaze axis. The processor 112 may furtheraccept input from the GPS unit 122, the gyroscope 120, and/or theaccelerometer 124 to determine the location and orientation of the HMD100. Subsequently, the processor 112 may control the user interface 115and the display panel 126 to display virtual images to the HMD wearerthat may include context-specific information based on the HMD locationand orientation as well as the HMD wearer's gaze axis.

HMD 100 could be configured as, for example, eyeglasses, goggles, ahelmet, a hat, a visor, a headband, or in some other form that can besupported on or from the wearer's head. Further, HMD 100 may beconfigured to display images to both of the wearer's eyes, for example,using two see-through displays. Alternatively, HMD 100 may include onlya single see-through display and may display images to only one of thewearer's eyes, either the left eye or the right eye. In otherembodiments, HMD 100 may include an opaque display configured to displayimages to a first eye or both eyes of the HMD wearer. In embodimentswhere an opaque display is presented to the first eye of the HMD wearer,a view of the real-world environment could be available to a second eyeof the HMD wearer.

A power supply 110 may provide power to various HMD components and couldrepresent, for example, a rechargeable lithium-ion battery. Variousother power supply materials and types known in the art are possible.

The function of the HMD 100 may be controlled by a processor 112 thatexecutes instructions stored in a non-transitory computer readablemedium, such as the memory 114. Thus, processor 112 in combination withinstructions stored in the memory 114 may function as a controller ofHMD 100. As such, processor 112 may control the user interface 115 toadjust what images are displayed by HMD 100. The processor 112 may alsocontrol the wireless communication interface 134 and various othercomponents of the HMD 100. The processor 112 may additionally representa plurality of computing devices that may serve to control individualcomponents or subsystems of the HMD 100.

In addition to instructions that may be executed by the processor 112,the memory 114 may store data that may include a set of calibratedwearer eye pupil positions and a collection of past eye pupil positions.Thus, the memory 114 may function as a database of information relatedto gaze direction. Such information may be used by HMD 100 to anticipatewhere the user will look and determine what images are to be displayedto the wearer. Calibrated wearer eye pupil positions may include, forinstance, information regarding the extents or range of the wearer's eyepupil movement (right/left and upwards/downwards) as well as wearer eyepupil positions that may relate to various reference axes.

Reference axes could represent, for example, an axis extending from aviewing location and through a target object or the apparent center of afield of view (i.e. the reference axis may correspond to a center lineof the field of view). Other possibilities for reference axes exist.Thus, a reference axis may further represent a basis for determiningdynamic gaze direction.

In addition, information may be stored in the memory 114 regardingpossible control instructions that may be enacted using eye movements.For instance, two consecutive wearer eye blinks may represent a controlinstruction directing the HMD 100 to capture an image with a peripheralcamera 140. Control instructions could also include the dwell-basedselection of a target object. For instance, if a wearer fixates visuallyupon a particular virtual image or real-world object for longer than apredetermined time period, a control instruction may be generated toselect the virtual image or real-world object as a target object. Manyother control instructions are possible.

In addition to the aforementioned features, memory 114 could storevarious recorded data from previous HMD/user interactions. For instance,multiple images of a HMD wearer's eye(s) could be averaged to obtain anaveraged eye gaze axis. This could lessen the effect of saccadic eyemovements or saccades, in which the eye moves in a rapid and somewhatrandom manner around an eye gaze axis. These saccades help humans buildup a mental image of a field of view with better resolution than if theeye remained static, and by averaging a number of eye images within aparticular time period, an average gaze axis could be determined withless saccadic ‘noise’.

Additionally, memory 114 could store recorded data regarding recent eyegaze axes for various application-based functions. For instance, therecent variance of the eye gaze axis could be coupled to scrollingimages generated by the HMD 100. In this embodiment, if recent eye gazeaxis variance is high, the images (e.g. text or other images) couldscroll faster. If the eye gaze axis variance is low, the images mayscroll slower or stop altogether. In this context, a lower variance ineye gaze axis could indicate the HMD wearer is concentrating on oneparticular gaze location, whereas a higher eye gaze axis variance meansthe opposite—the HMD wearer may be quickly scanning a document anddesire a faster scrolling speed.

Depending on the content that is presented on the HMD display, thevariance may differ depending on the axis along which it is measured.For example, the horizontal variance of a HMD wearer's eye gaze may behigh while the vertical variance may be relatively low. This couldindicate to the HMD 100 that the wearer is reading text. Accordingly,text scrolling/tracking could be adjusted in a different or morecontrolled fashion compared to ‘non-reading’scrolling/panning/pagination situations.

The HMD 100 may include a user interface 115 for providing informationto the wearer or receiving input from the wearer. The user interface 115could be associated with, for example, the displayed virtual images, atouchpad, a keypad, buttons, a microphone, and/or other peripheral inputdevices. The processor 112 may control the functioning of the HMD 100based on input received through the user interface 115. For example, theprocessor 112 may utilize user input from the user interface 115 tocontrol how the HMD 100 displays images within a field of view ordetermine what images the HMD 100 displays.

The infrared camera 116 may be utilized by the eye-tracking system 102to capture images of a viewing location associated with the HMD 100.Thus, the infrared camera 116 may image the eye of a HMD wearer that maybe located at the viewing location. The images could be either videoimages or still images. The images obtained by the infrared camera 116regarding the HMD wearer's eye may help determine where the wearer islooking within the HMD field of view, for instance by ascertaining thelocation of the HMD wearer's eye pupil. Analysis of the images obtainedby the infrared camera 116 could be performed by the processor 112 inconjunction with the memory 114.

The imaging of the viewing location could occur continuously or atdiscrete times depending upon, for instance, user interactions with theuser interface 115. The infrared camera 116 could be integrated into theoptical system 106. Alternatively, the infrared camera 116 could bemounted separately from the optical system 106 and/or HMD 100.Furthermore, the infrared camera 116 could additionally represent avisible light camera with sensing capabilities in the infraredwavelengths.

The infrared light source 118 could represent one or more infraredlight-emitting diodes (LEDs) or infrared laser diodes that mayilluminate a viewing location. Thus, one or both eyes of a wearer of theHMD 100 may be illuminated by the infrared light source 118. Theinfrared light source 118 may be positioned along an optical axis commonto the infrared camera, and/or the infrared light source 118 may bepositioned elsewhere. The infrared light source 118 could be mountedseparately from the optical system 106 and/or HMD 100. The infraredlight source 118 may illuminate the viewing location continuously or maybe turned on at discrete times. Additionally, when illuminated, theinfrared light source 118 may be modulated at a particular frequency.

The HMD-tracking system 104 could be configured to provide a HMDposition and HMD orientation to the processor 112. This position andorientation data may help determine a reference axis to which a gazeaxis is compared. For instance, the reference axis may correspond to theorientation of the HMD.

The gyroscope 120 could be a microelectromechanical system (MEMS)gyroscope or a fiber optic gyroscope. The gyroscope 120 may beconfigured to provide orientation information to the processor 112. TheGPS unit 122 could be a receiver that obtains clock and other signalsfrom GPS satellites and may be configured to provide real-time locationinformation to the processor 112. The HMD-tracking system 104 couldfurther include an accelerometer 124 configured to provide motion inputdata to the processor 112.

The optical system 106 could represent components configured to providevirtual images to a viewing location. An example of optical system 106is described in detail below.

Various peripheral devices 108 may be included in the HMD 100 and mayserve to provide information to and from a wearer of the HMD 100. In oneexample, the HMD 100 may include a wireless communication interface 134for wirelessly communicating with one or more devices directly or via acommunication network. For example, wireless communication interface 134could use 3 G cellular communication, such as CDMA, EVDO, GSM/GPRS, or4G cellular communication, such as WiMAX or LTE. Alternatively, wirelesscommunication interface 134 could communicate with a wireless local areanetwork (WLAN), for example, using WiFi. In some embodiments, wirelesscommunication interface 134 could communicate directly with a device,for example, using an infrared link, Bluetooth, or ZigBee.

Although FIG. 1 shows various components of the HMD 100 (i.e., wirelesscommunication interface 134, processor 112, memory 114, infrared camera116, display panel 126, GPS 122, and user interface 115) as beingintegrated into HMD 100, one or more of these components could bephysically separate from HMD 100. For example, infrared camera 116 couldbe mounted on the wearer separate from HMD 100. Thus, the HMD 100 couldbe part of a wearable computing device in the form of separate devicesthat can be worn on or carried by the wearer. The separate componentsthat make up the wearable computing device could be communicativelycoupled together in either a wired or wireless fashion.

FIG. 2 illustrates a top view of an optical system 200 that isconfigured to display a virtual image superimposed upon a real-worldscene viewable along a viewing axis 204. For clarity, a distal portion232 and a proximal portion 234 represent optically-coupled portions ofthe optical system 200 that may or may not be physically separated. Anexample embodiment includes a display panel 206 that may be illuminatedby a light source 208. Light emitted from the light source 208 isincident upon the distal beam splitter 210. The light source 208 mayinclude one or more light-emitting diodes (LEDs) and/or laser diodes.The light source 208 may further include a linear polarizer that acts topass one particular polarization to the rest of the optical system.

In an example embodiment, the distal beam splitter 210 is a polarizingbeam splitter that reflects light depending upon the polarization oflight incident upon the beam splitter. To illustrate, s-polarized lightfrom the light source 208 may be preferentially reflected by a distalbeam-splitting interface 212 towards the display panel 206. The displaypanel 206 in the example embodiment is a liquid crystal-on-silicon(LCOS) display, but could also be a digital light projector (DLP)micro-mirror display, or other type of reflective display panel. Thedisplay panel 206 acts to spatially-modulate the incident light togenerate a light pattern. Alternatively, the display panel 206 may be anemissive-type display such as an organic light-emitting diode (OLED)display.

In the example in which the display panel 206 is a LCOS display panel,the display panel 206 generates a light pattern with a polarizationperpendicular to the polarization of light initially incident upon thepanel. In this example embodiment, the display panel 206 convertsincident s-polarized light into a light pattern with p-polarization. Thegenerated light pattern from the display panel 206 is directed towardsthe distal beam splitter 210. The p-polarized light pattern passesthrough the distal beam splitter 210 and is directed along an opticalaxis 214 towards the proximal region of the optical system 200. In anexample embodiment, the proximal beam splitter 216 is also a polarizingbeam splitter. The light pattern is at least partially transmittedthrough the proximal beam splitter 216 to the image former 218. In anexample embodiment, image former 218 includes a concave mirror 230 and aproximal quarter-wave plate 228. The light pattern passes through theproximal quarter-wave plate 228 and is reflected by the concave mirror230.

The reflected light pattern passes back through proximal quarter-waveplate 228. Through the interactions with the proximal quarter-wave plate228 and the concave mirror 230, the light patterns are converted to thes-polarization and are formed into a viewable image. This viewable imageis incident upon the proximal beam splitter 216 and the viewable imageis reflected from proximal beam splitting interface 220 towards aviewing location 222 along a viewing axis 204. A real-world scene isviewable through a viewing window 224. The viewing window 224 mayinclude a linear polarizer in order to reduce stray light within theoptical system. Light from the viewing window 224 is at least partiallytransmitted through the proximal beam splitter 216. Thus, both a virtualimage and a real-world image are viewable to the viewing location 222through the proximal beam splitter 216.

Although FIG. 2 depicts the distal portion 232 of the optical systemhousing as to the left of the proximal portion 234 of the optical systemhousing when viewed from above, it is understood that other embodimentsare possible to physically realize the optical system 200, including thedistal portion 232 being configured to be to the right, below and abovewith respect to the proximal portion 234. Further, although an exampleembodiment describes an image former 218 as comprising a concave mirror230, it is understood by those skilled in the art that the image former218 may comprise a different optical element, such as an optical lens ora diffractive optic element.

In one embodiment, the proximal beam splitter 216, the distal beamsplitter 210, and other components of optical system 200 are made ofglass. Alternatively, some or all of such optical components may bepartially or entirely plastic, which can also function to reduce theweight of optical system 200. A suitable plastic material is Zeonex®E48R cyclo olefin optical grade polymer which is available from ZeonChemicals L.P., Louisville, Ky. Another suitable plastic material ispolymethyl methacrylate (PMMA).

An example embodiment may include an infrared light source 226 that isconfigured to illuminate the viewing location 222. Although FIG. 2depicts the infrared light source 226 as adjacent to viewing window 224,those skilled in the art will understand that the infrared light source226 could be located elsewhere, such as on the side of the proximal beamsplitter 216 that is adjacent to the viewing location 222 or in thedistal portion 232 of the optical system 200. The infrared light source226 may represent, for example, one or more infrared light-emittingdiodes (LEDs). Infrared LEDs with a small size may be implemented, suchas the Vishay Technology TSML 1000 product.

Further, those skilled in the art will understand that, for besteye-tracking accuracy, it may be advantageous to obtain infrared imagesof the eye pupil using light sources that illuminate the eye frompositions off-axis and/or on-axis with respect to the viewing axis 204.Therefore, the infrared light source 226 may include one or more LEDslocated at different locations in the optical system 200.

Infrared light generated from the infrared light source 226 isconfigured to be incident upon the viewing location 222. Thus, thewearer's eye pupil may be illuminated with the infrared light. Theinfrared light may be reflected from the wearer's eye back along theviewing axis 204 towards the proximal beam splitter 216. A portion ofthe reflected infrared light may be reflected from the beam splittinginterface 220 towards the image former 218.

In order to transmit infrared light to an infrared camera 202, the imageformer 218 may include a dichroic thin film configured to selectivelyreflect or transmit incident light depending upon the wavelength of theincident light. For instance, the dichroic thin film may be configuredto pass infrared light while reflecting visible light. In an exampleembodiment, the visible light pattern generated by the display panel 206may be reflected by the concave mirror 230 and the visible light patternmay be formed into a viewable image. The infrared light may thus bepreferably transmitted through the concave mirror 230 to infrared camera202. Dichroic thin film coatings are available commercially fromcompanies such as JML Optical Industries and Precision Glass & Optics(PG&O) and comprise multiple layers of dielectric and/or metal films.These dichroic coatings are also called ‘cold mirrors’.

In an example embodiment, a small aperture or apertures may beintroduced into the image former 218, which may be realized by one ormore pinholes in the concave mirror 230. In this example embodiment,most of the visible and infrared light is reflected off of and formed bythe image former 218 into an image viewable by the HMD wearer. Some ofthe visible and infrared light passes through the aperture and isincident upon the infrared camera 202. The infrared camera 202 mayselectively filter and detect the infrared light from the combination ofvisible and infrared light to obtain information regarding the wearer'seye pupil location. Alternatively, the infrared light source 226 may bemodulated to provide a frequency reference for a lock-in amplifier orphase-locked loop in order that the infrared light signal is obtainedefficiently. Also, the visible light source 208 may be modulated andinfrared light detection could be performed when the visible lightsource 208 is off, for example. Those with skill in the art willunderstand that there are other variations of transducing an infraredlight signal mixed with a visible light signal with an infrared cameraand that those variations are included implicitly in this specification.

FIG. 3A presents a front view of a head-mounted display (HMD) 300 in anexample embodiment that includes a head-mounted support 309. FIGS. 3Band 3C present the top and side views, respectively, of the HMD in FIG.3A. Although this example embodiment is provided in an eyeglassesformat, it will be understood that wearable systems and HMDs may takeother forms, such as hats, goggles, masks, headbands and helmets. Thehead-mounted support 309 includes lens frames 314 and 316, a centerframe support 318, lens elements 310 and 312, and extending side-arms320 and 322. The center frame support 318 and side-arms 320 and 322 areconfigured to secure the head-mounted support 309 to the wearer's headvia the wearer's nose and ears, respectively. Each of the frame elements314, 316, and 318 and the extending side-arms 320 and 322 may be formedof a solid structure of plastic or metal, or may be formed of a hollowstructure of similar material so as to allow wiring and componentinterconnects to be internally routed through the head-mounted support309. Alternatively or additionally, head-mounted support 309 may supportexternal wiring. Lens elements 310 and 312 are at least partiallytransparent so as to allow the wearer to look through them. Inparticular, the wearer's left eye 308 may look through left lens 312 andthe wearer's right eye 306 may look through right lens 310. Opticalsystems 302 and 304, which may be configured as shown in FIG. 2, may bepositioned in front of lenses 310 and 312, respectively, as shown inFIGS. 3A, 3B, and 3C. Optical systems 302 and 304 may be attached to thehead-mounted support 309 using support mounts 324 and 326, respectively.Furthermore, optical systems 302 and 304 may be integrated partially orcompletely into lens elements 310 and 312, respectively.

Although this example includes an optical system for each of thewearer's eyes, it is to be understood that a HMD might include anoptical system for only one of the wearer's eyes (either left eye 308 orright eye 306 ). As described in FIG. 2, the HMD wearer maysimultaneously observe from optical systems 302 and 304 a real-worldimage with an overlaid virtual image. The HMD 300 may include variouselements such as a processor 340, a touchpad 342, a microphone 344, anda button 346. The computer 340 may use data from, among other sources,various sensors and cameras to determine the virtual image that shouldbe displayed to the user. In an example embodiment, as describedearlier, an infrared light source or sources may illuminate the viewingposition(s) 308 and 306, i.e. the wearer's eye(s), and the reflectedinfrared light may be preferentially collected with an infrared camera.

Those skilled in the art would understand that other user input devices,user output devices, wireless communication devices, sensors, andcameras may be reasonably included in such a wearable computing system.

FIGS. 4A and 4B depict side and front views of an eye as well asschematic drawings of pupil location information under differentconditions. One way to determine a gaze axis of a person is to ascertainthe position of the person's eye pupil with respect to a referencepoint, such as a viewing location. To track eye pupil movements,infrared light is generally reflected off of a person's eye. Thereflected light may be collected and detected with an infrared detector.Upon imaging of the eye, image processing can be conducted with aprocessor 112 in order to determine, for instance, the extents andcentroid location of the person's pupil. The other known means andmethods of eye-tracking, including the use of visible light illuminationand/or imaging techniques are possible.

For example, in an embodiment 400, a person may be looking directlyforward as depicted in FIG. 4A. The eye 412 is open and the pupil 418 islocated along a reference axis 410. After image processing, which mayinclude edge detection, the position of the pupil may be determined tobe at pupil location 422. In this example, the processor 112 maysubsequently determine that the gaze axis based on the pupil location422 coincides with a reference axis 410. Virtual image display positionand movement may be adjusted due to the determined pupil location 422.For instance, the processor 112 may adjust a tracking rate to zero whenthe gaze axis and the reference axis are equivalent or nearlyequivalent. This may allow a user to slowly read critical text orclosely examine a virtual image, for example.

In an example embodiment 424, as illustrated in FIG. 4B, a person may belooking upwards with respect to a reference axis 428. The eye 434 isopen and the pupil location is generally higher than a reference point440. In this situation, imaging the person's pupil 438 with infraredlight may result in a determined pupil position 442. The processor 112may determine that the gaze axis 430 that is above the reference axis428. The angle difference 432 may represent the absolute difference inangle between the reference axis 428 and the gaze axis 430. Theprocessor 112 may calculate the angle difference 432 and, based on theangle difference 432, adjust a tracking rate. For instance, a largeangle difference 432 could represent an adjustment in tracking rate suchthat the tracking rate is higher, for instance to scroll a virtual imageacross a field of view at a faster rate.

Other embodiments could include the use of different eye gazedetermination techniques. For instance, instead of using the eye pupilto determine gaze axis, it is possible to track eye motions using theboundary between the sclera and iris (416 and 436 in FIGS. 4A and 4B).For the purposes of determining an eye gaze axis, finding the centroidof the sclera/iris boundary may be equivalent to finding the centroid ofa pupil.

3. A Method for Adjusting Virtual Images within a Field of View Based ona Gaze Axis, a Reference Axis, and a Tracking Rate.

A method 500 is provided for adjusting virtual images within a field ofview based on a gaze axis, a reference axis and a tracking rate. Method500 could be performed using an HMD that is configured as shown in anyof FIGS. 1-3C or configured in some other way. FIG. 5 illustrates thesteps in an example method, however, it is understood that in otherembodiments, the steps may appear in different order and steps may beadded or subtracted.

In the method, a gaze axis is determined within a field of view of ahead-mounted display (HMD) (Step 502 ). The HMD is generally configuredto display virtual images to be viewable at a viewing location and couldbe an HMD similar to an aforementioned embodiment. The gaze axis couldbe determined to be an axis extending from a center of the HMD wearer'spupil. The gaze axis could be similar to the gaze axis 430 depicted inFIG. 4B. The pupil location could be determined using eye glint imagesor by other eye-tracking techniques detailed above.

A reference axis related to the HMD may also be determined in the method(Step 504 ). The reference axis could be determined by the processor 112based on the orientation of the HMD and may correspond to the apparentcenter of the HMD field of view, for example. Other reference axes arepossible.

A tracking rate related to the movement of virtual images may beadjusted based on the gaze axis and the reference axis (Step 506 ). Thetracking rate may be the rate at which virtual images are panning and/orscrolling across the HMD field of view. The tracking rate may dependupon the data that is displayed. For instance, words on a document mayscroll in a vertical fashion from the bottom to the top of the HMD fieldof view to simulate reading downwards along a printed page. In thiscase, the display may move the text upwards at a tracking rate of around1 second per line.

A tracking rate could also be related to the motion of a HMD wearer. Forinstance, when displaying virtual images that may be user interfacemenus, for instance, the HMD may attempt to base the tracking rate ofthe virtual images on the rate of HMD movement. More specifically, anHMD wearer may access a user interface by changing the orientation ofthe HMD (for instance rotating one's head and/or body to selectdifferent elements of the user interface). Thus, the user interfacecould be at least partially anchored to locations and objects in thereal world and the virtual images could be adjusted or panned at a rateproportional to the rate of change of the HMD orientation.

Further, the tracking rate could be adjusted by the angle differencebetween the gaze axis and the reference axis. For instance, the trackingrate could be increased if the angle difference between the gaze axisand the reference axis is large and the tracking rate could be decreasedif the angle difference is small. The tracking rate could also beadjusted based on the direction of the vector between the reference axisand the gaze axis. More detailed examples are given below.

The virtual images may be adjusted within the field of view based on thegaze axis, the reference axis, and the tracking rate (Step 508 ). If thetracking rate is adjusted lower, the movement rate of the virtual imagesmay slow, for instance. Conversely, with a higher tracking rate, thevirtual images may appear to move more quickly within the field of view.

FIGS. 6A, 6B, 6C, and 6D illustrate an example in which a determinedgaze axis controls text scrolling. In the example embodiment 600,virtual images including text are presented within a field of view 602.The text 604 may be scrolling slowly upwards at a normal tracking ratesimilar in fashion to credits at the end of a movie (around one line persecond, for instance). A gaze point 606 may be ascertained related to agaze axis and thus to the position of an eye pupil of a wearer of anHMD. In this example embodiment, the reference axis may be considered asoriginating from the wearer's eye and going through the apparent centerof the field of view 602. When a wearer is reading normally and theangle between the reference axis and the gaze axis is relatively small,the tracking rate may stay unchanged and the text 604 may continue toscroll upwards.

In FIG. 6B, the wearer of the HMD may move his or her gaze point from acentral location 610 to a location near the bottom of the field of view612, as shown in a particular field of view 608. When this change in eyegaze point is detected by the processor 112, the processor 112 mayadjust the tracking rate of the virtual images to increase the trackingrate, such as illustrated in field of view 614. In particular, theprocessor 112 may determine that the wearer is reading quickly and tryto supply more text by increasing the tracking rate. Thus, the upwardmovement rate of text 604 may increase.

In a related scenario 616 depicted in FIG. 6C, while text 604 isscrolling upward within a HMD field of view 602, the wearer of the HMDmay move his or her gaze point from a central location 610 to a location618 near the top of the field of view 602. When this change in eye gazepoint is detected by the processor 112, the processor 112 may adjust thetracking rate of the virtual images to decrease the tracking rate. Inparticular, the processor 112 may determine that the wearer is readingslowly and try to supply text to the reader's eye more slowly bydecreasing the tracking rate. Thus, the upward movement rate of text 604may decrease.

FIG. 6D depicts a scenario 620 wherein a tracking rate may be adjustedto zero. For example, while text 604 is scrolling upward within a HMDfield of view 602, the wearer of the HMD may move his or her gaze pointto a central location 622 of the field of view 602, as shown in aparticular field of view 616. Further, the wearer of the HMD may fixatehis or her eye gaze point upon the central location 622 for somepredetermined period of time. When this eye gaze point position and/orthe eye gaze point fixation is detected by the processor 112, theprocessor 112 may adjust the tracking rate of the virtual images tofurther decrease or zero the tracking rate. In particular, the processor112 may determine that the wearer wants to focus on a particular elementof the virtual image and may provide a more stable virtual image bydecreasing or zeroing the tracking rate. Thus, the movement of text 604may decrease further in rate or stop completely. Furthermore, gaze axismovements near the reference axis may provide for smaller tracking ratesthan gaze axis movements at larger angle differences. Thus, eye gazemovements around a reference axis may provide finer virtual imagetracking control while gazing farther away from the reference axis mayprovide coarse virtual image panning and scrolling tracking control.

FIGS. 6A, 6B, 6C, and 6D illustrate an example embodiment in which anHMD may adjust text scrolling while the HMD could be stationary.However, example embodiments in which the HMD adjusts virtual imageswhile simultaneously translating and/or rotating may also be considered.For instance, a HMD may display a user interface in which the virtualimages are substantially anchored to the real-world environment. This‘world-fixed’ user interface could appear to the HMD user as though thevirtual images of the user interface are substantially fixed to aninside surface of an imaginary ring that surrounds the user's head. Thatis, when a HMD user turns his or her head to the left, the virtualimages rotate within his or her field of view to the right, and viceversa.

In one situation, the HMD user may access and navigate menus and iconsin the user interface by moving the HMD and by using the aforementionedHMD reference axis as a pointing device or cursor. However, accessingthis menu may be complicated if the HMD user is in motion. For instance,when travelling on a subway, the HMD user may round a corner, which mayaffect the HMD position as well as orientation. Thus, a user interfacemenu controlled only by the HMD position and/or orientation may produceerrors when the HMD changes its relative reference position such asinadvertent menu selection or rotation of icons in the viewable userinterface.

In an example embodiment, the utilization of an eye-tracking systemcould reduce inadvertent movement of the virtual images due to changesin HMD position and orientation. For instance, if the HMD positionand/or orientation changes but the eye-tracking system detects nocorresponding eye movement (anticipatory eye gaze changes, for example),the HMD may be configured to not adjust the virtual images with respectto the changing HMD position.

The example embodiment 700 is illustrated in FIG. 7A in which the HMDuser is riding a subway forward (position 704 to position 706 ) and thenthe subway curves left (position 706 to position 708 ). In the exampleembodiment 712, the HMD user may be accessing a user interface menu thatcould include icons and/or menus associated with files 718, photos 720,e-mail 722, contacts 724 and a calendar 726. FIG. 7B illustrates apossible view that the HMD user may see when located at position 704.The e-mail menu 722 could be centered within the HMD display 714 and thegaze point 710 could be determined to be near the center of the HMDdisplay 714.

As described above, the user interface icons and/or menus could bearranged in imaginary ring that surrounds the HMD user's head. In theexample embodiment 712, if at rest, the user interface may be configuredto remain substantially ‘world-fixed’ and rotate in the oppositedirection at least due to HMD panning and eye gaze change. For instance,if the HMD user rotates the HMD to the right and gazes towards the rightside of the HMD display 714, the user interface menus may rotate to theleft, allowing the contacts 724 and calendar 726 to be displayed in theHMD display 714.

However, as shown in FIG. 7C, displacements, such as moving straightahead may not necessarily create a change in the displayed objects. Forinstance, if the HMD user moves forward in the subway car from position704 to position 706, the menu may stay substantially the same.Alternatively, displacements in HMD location could represent inputs thatmay cause the displayed objects to be adjusted. For instance, physicalmovements of the HMD could represent ‘walking’ through the userinterface in three-dimensional space.

FIG. 7D illustrates an embodiment 732 where the HMD user may betravelling on the subway car at position 708. At that position 708, theHMD user is traveling forward and also rotating gradually to the left.If the user interface is not corrected by an eye-tracking system, thecontroller may interpret the HMD movement as an intentional movement toturn left. In general, this motion may lead to rotating the menus to theright. In this case, the photos 720 menu may be moved towards the centerof the HMD display 716.

However, FIG. 7E illustrates an embodiment 736 that may use aneye-tracking system to correctly determine the actual desired speed ofuser interface movement. In this case, the HMD user is moving throughposition 708, while moving forward and rotating to the left. At the sametime, an eye gaze point 738 may be determined to remain at the center ofthe HMD display 716. The eye gaze fixation may cause the menu to notrotate.

Other embodiments of adjusting the tracking rate of virtual images in anHMD system are possible and are not meant to be limited by the abovediscussion. Those skilled in the art will understand that HMDs maypresent many different types of information in the form of virtualimages to a wearer. Accordingly, each of these various virtual imagesmay be assigned a different tracking rate and vector, which may be basedon at least one or more context-related factors such as wearer readingspeed, HMD orientation, HMD motion, HMD location, gaze axis, etc.

A further example of how gaze direction may be used to correct formotion of the HMD is illustrated by the pseudo-code set forth below inTable 1. The routine may be called every time that there is a new sensorreading relating to the orientation of the HMD, for example, a sensorreading from gyroscope 120, accelerometer 124, or other component ofHMD-Tracking system 104. The routine is able to calculate two variables,adjustedX and adjustedY, which relate to how far the displayed imagesare to be moved in the x and y directions, respectively, using themoveScreen (adjustedX, adjustedY) function. Specifically, the variablesdx and dy, which represent the distances that the wearer's head hastravelled in the x and y directions, respectively, are determined fromthe sensor data. The variables vx and vy, which represent the x and ydisplacements between the wearer's gaze location on the screen and thecenter of the screen, may be calculated based on the angle differencebetween the gaze axis and a reference axis that goes through the centerof the screen. The variable adjustedX may then be calculated as afunction of dx and vx, and the variable adjustedY may be calculated as afunction of dy and vy. In this way, the wearer of the HMD is able tokeep the position of the displayed images fixed, notwithstanding motionof the HMD, by gazing at the center of the screen.

TABLE 1 var radiusX = adjustable number of units; var radiusY =adjustable number of units; // callback method that is called every timethere is a new sensor reading. function handleHeadMovement( )  { //calculate the distance the head has moved.  This distance might // befaulty due to sensor drift or unintended movement of the wearer, // aswhen riding the subway or walking around a corner var dx = relativedistance that the head travelled on the x axis; var dy = relativedistance that the head travelled on the y axis; // calculate thevariance of the eye gaze from the center of the screen var vx = distancethe eye gaze is from the center of the screen on the x axis; var vy =distance the eye gaze is from the center of the screen on the y axis; //adjust the distance travelled based on the distance the eye gaze is from// the center.  Make sure that the distance travelled is onlydecremented, // never augmented var adjustedX = dx * Math.min( 1, vx /radiusX ); var adjustedY = dy * Math.min( 1, vy / radiusY ); // move thescreen to reflect the adjusted movement. moveScreen( adjustedX,adjustedY ); }

4. Non-Transitory Computer Readable Medium to Determine Speed of ImageMovement Using Eye Gaze Detection.

Some or all of the functions described above in method 500 andillustrated in FIGS. 5, 6A, 6B, 6C, 6D, 7A, 7B, 7C, 7D, and 7E may beperformed by a computing device in response to the execution ofinstructions stored in a non-transitory computer readable medium. Thenon-transitory computer readable medium could be, for example, a randomaccess memory (RAM), a read-only memory (ROM), a flash memory, a cachememory, one or more magnetically encoded discs, one or more opticallyencoded discs, or any other form of non-transitory data storage. Thenon-transitory computer readable medium could also be distributed amongmultiple data storage elements, which could be remotely located fromeach other. The computing device that executes the stored instructionscould be a wearable computing device, such as a wearable computingdevice 100 illustrated in FIG. 1. Alternatively, the computing devicethat executes the stored instructions could be another computing device,such as a server in a server network. A non-transitory computer readablemedium may store instructions executable by the processor 112 to performvarious functions. For instance, instructions that could be used tocarry out method 500 may be stored in memory 114 and could be executedby processor 112. In such an embodiment, upon receiving gaze informationfrom the eye-tracking system 102, the processor 112 carry outinstructions to determine a gaze axis and a reference axis as well as tocontrol the HMD 100 to display virtual images within the HMD field ofview and adjust a tracking rate based on the gaze axis and the referenceaxis.

CONCLUSION

The above detailed description describes various features and functionsof the disclosed systems, devices, and methods with reference to theaccompanying figures. While various aspects and embodiments have beendisclosed herein, other aspects and embodiments will be apparent tothose skilled in the art. The various aspects and embodiments disclosedherein are for purposes of illustration and are not intended to belimiting, with the true scope and spirit being indicated by thefollowing claims.

1-10. (canceled)
 11. A method, comprising: determining a gaze axiswithin a field of view of a head-mounted display (HMD), wherein the HMDis configured to display virtual images within the field of view;determining a reference axis related to the HMD; adjusting a trackingrate based on the gaze axis and the reference axis; and controlling theHMD to move the virtual images within the field of view based on thetracking rate.
 12. The method of claim 11, wherein determining the gazeaxis comprises: obtaining at least one image of at least one eye of awearer of the HMD; and determining the gaze axis from the at least oneimage.
 13. The method of claim 12, wherein the at least one imagefurther comprises at least one image of at least one eye pupil of awearer of the HMD.
 14. The method of claim 11, wherein the referenceaxis related to the HMD corresponds to a central axis of the field ofview.
 15. The method of claim 11, wherein the reference axis related tothe HMD comprises an axis that extends from the viewing location to atarget object.
 16. The method of claim 11, wherein the reference axisrelated to the HMD corresponds to an orientation of the HMD.
 17. Themethod of claim 11, wherein adjusting the tracking rate furthercomprises: determining an angle difference between the gaze axis and thereference axis; and adjusting the tracking rate based on the angledifference.
 18. The method of claim 17, wherein adjusting the trackingrate further comprises adjusting the tracking rate proportionally basedon the angle difference.
 19. The method of claim 11, wherein controllingthe HMD to move the virtual images within the field of view comprisescontrolling the HMD to pan the virtual images within the field of view.20. The method of claim 11, wherein the virtual images compriseone-dimensional virtual objects.
 21. The method of claim 11, wherein thevirtual images comprise two-dimensional virtual objects.
 22. Anon-transitory computer readable medium having stored thereininstructions executable by a computing device to cause the computingdevice to perform functions, comprising: determining a gaze axis withina field of view of a head-mounted display (HMD), wherein the HMD isconfigured to display virtual images within the field of view;determining a reference axis related to the HMD; adjusting a trackingrate based on the gaze axis and the reference axis; and controlling theHMD to move the virtual images within the field of view based on thetracking rate.
 23. The non-transitory computer readable medium of claim22, wherein determining a reference axis comprises receiving informationabout a target object.
 24. The non-transitory computer readable mediumof claim 22, wherein determining a reference axis comprises receivingglobal positioning system (GPS) data.
 25. The non-transitory computerreadable medium of claim 22, wherein determining a reference axiscomprises receiving accelerometer data.
 26. The non-transitory computerreadable medium of claim 22, wherein determining a reference axiscomprises receiving gyroscope data.
 27. The non-transitory computerreadable medium of claim 22, wherein adjusting the tracking ratecomprises adjusting the tracking rate proportionally based on an angledifference between the gaze axis and the reference axis.
 28. Thenon-transitory computer readable medium of claim 22, wherein controllingthe HMD to move the virtual images within the field of view comprisescontrolling the HMD to pan the virtual images within the field of view.29. The non-transitory computer readable medium of claim 22, wherein thevirtual images comprise one-dimensional virtual objects.
 30. Thenon-transitory computer readable medium of claim 22, wherein the virtualimages comprise two-dimensional virtual objects.